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PATENT APPLICATION 
IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re application of 

Reinhard JANSSEN, Frank LINDNER 
and Berthold ULMER 

Appln . No . : 

Filed: May 14, 2001 

For: METHOD FOR INCREASING THE POWER OF A 
TRAFFIC SIGN RECOGNITION SYSTEM 

Attorney Docket No.: 3926,027 

PRELIMINARY AMENDMENT 

Box : PCT 

Honorable Commissioner of 

Patents and Trademarks 
Washington, D.C. 20231 

Sir: 

Prior to examination of the above -identified application, 
please amend the application as follows: 

IN THE SPECIFICATION : 

Page 1, before paragraph [0001] insert: 
--BACKGROUND OF THE INVENTION 

Field of the Invention- - 

Page 1, paragraph [0001] delete "according to the 
precharacterizing portion of Patent Claims 1 and 10". 

Page 1, before paragraph [0002] insert: 
--Description of the Related Art--; 

Page 1, before paragraph [0003] insert: 

- -SUMMARY OF THE INVENTION- - 

Page 1, before paragraph [0004] insert: 

--DETAILED DESCRIPTION OF THE INVENTION--. 
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IN THE CLAIMS: 

Page 9, line 1, please delete "PATENT CLAIMS" and insert 
therefore --What is Claimed is:-- 

Please cancel original Claims 1-23 and PCT Substitute Claims 
1-23 and add the following new Claims 24-49 as follows: 

--24. A process for improving performance of a system for 
recognizing traffic signs, said system including a camera and a 
therewith associated evaluation unit for image recognition or 
image display, said process comprising: 

utilizing information originating from at least one of 

(a) a map-based navigation system and 

(b) a traffic information system 

in the evaluation or display of contents of traffic signs. 

25. A process according to Claim 24, wherein the system for 
recognition and/or display of traffic signs is in communication 
with the map-based navigation system or the traffic information 
system via an onboard data bus. 

26. A process according to Claim 24, wherein the system for 
traffic sign recognition is capable of operating at a normal 
performance level and at least one enhanced performance level, 
and wherein said system is caused to operate at said enhanced 
performance level when it is determined on the basis of map-based 
information that the vehicle is passing through an area in which 
recognition of traffic signs may be problematic. 

27. A process according to Claim 26, wherein within said 
problematic areas supplemental subdivided recognition processes 
are carried out, comprising: 
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a) when entering or leaving communities, carrying out a 
specific search for signs indicating entry of a community 
and leaving of a community, 

b) when in the area of traffic influencing facilities, 
changing signs or traffic lights, searching for any change 
in the type and manner of the representation of the traffic 
signs, 

c) when in an area in which poor visibility due to fog or 
rain may be present and a higher probability of contrast - 
poor images may be required, effecting an increase in the 
contrast enhancement of the image data. 

28. A process according to Claim 24, wherein thereby 
characterized, that the type or the position or the condition or 
the visibility or the size of a recognized traffic sign is stored 
in a data storage unit associated with the map-based navigation 
system or the traffic information system. 

29. A process according to Claim 28, further comprising 
emitting, by means of a signal producer contained in the display 
unit, an acoustic or optical signal when along an already 
traveled road segment a change with respect to the position or a 
change as to the presence or absence of a traffic sign occurs . 

30. A process according to Claim 24, wherein the type and/or the 
position and/or the condition and/or the visibility and/or the 
size of a recognized traffic sign is stored in a data storage 
unit associated with the system for recognition of traffic signs. 

31. A process according to Claim 30, further comprising 
emitting, by means of a signal producer contained in the display 
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unit, an acoustic or optical signal when along an already 
traveled road segment a change with respect to the position or a 
change as to the presence or absence of a traffic sign occurs. 

32. A process according to Claim 24, wherein starting at the 
map-based navigation system or the traffic information system, a 
program for application of a decision logic is controlled in such 
a manner, that the image of the traffic sign is displayed for 
such a period, until a predetermined traveled path stored in the 
storage unit has been traveled. 

33. A process according to claim 32, wherein said predetermined 
traveled path is dependent upon the class of traffic sign. 

34. A process according to Claim 32, wherein said predetermined 
traveled paths are programmed into a data storage unit. 

35. A process according to Claim 24, wherein the system for 
recognition of traffic signs, during the processing of image data 
provided by an image sensor, 

examines these image data for the presence of traffic signs, 

extracts data corresponding to traffic signs, 

separates the extracted data into upper and lower classes 

and in association therewith extracts class-specific 

characterizing data, and 

supplies these extracted class-specific characterizing data 

to a separate classification. 

36. A process according to Claim 35, wherein the classification 
occurs hierarchically, in multiple classification steps. 
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37. A process according to Claim 35, wherein the classification 
steps are essentially set up follows: 

a) first, only the characterizing data associated 

with the upper class of the object is supplied to a 

classifier, 

aa) upon successful classification, wherein the class 
is recognized correctly with a high degree of 
confidence, the characterizing data are replaced by the 
appropriate symbolic representation of the upper class 
stored in the memory unit, 

ab) upon unsuccessful classification, wherein the 
class could not be correctly recognized with a high 
degree of confidence, the characterizing data for the 
upper class and the charactering data for the lower 
class are replaced by the corresponding original image 
data originating from the image sensor, 

b) if the classification of the upper class was 
successful, then subsequently the characterizing data 
associated with the lower class of the object are supplied 
to a classifier, 

ba) upon successful classification, wherein the class 
is recognized with a high degree of confidence, the 
characterizing data are replaced by the appropriate 
symbolic representation of the lower class stored in 
the memory unit, 

bb) upon unsuccessful classification, wherein the 
class could not be correctly recognized with a high 
degree of confidence) the characterizing data for the 
lower class are substituted by the original image data 
from the image sensor. 
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38. A vehicle -mounted device for enhancing the performance of a 
system for recognizing traffic signs, which system includes a 
camera and a therewith associated evaluation unit for image 
recognition or for image display, 

wherein said system is associated with at least one of (a) a 
map-based navigation system and (b) a traffic information system 
so as to utilize information originating therefrom in the process 
of recognition or display of contents of traffic signs. 

39. A device according to Claim 38, wherein the system, the map- 
based navigation system, and/or the vehicle traffic information 
system are connected via an onboard data bus. 

40. A device according to Claim 38, wherein the system for 
traffic sign recognition has the capacity to work at a base level 
of performance and an enhanced level of performance, and wherein 
said enhanced level of performance is engaged when, on the basis 
of map-based information, it is determined that the vehicle is 
passing through an area in which the recognition of traffic signs 
may be problematic. 

41. A device according to Claim 38, wherein one of (a) the map- 
based navigation system and (b) the traffic information system is 
provided with a data storage unit, in which at least one of the 
type, the position, the condition, the visibility and the size of 
the recognized traffic sign is stored. 

42. A device according to Claim 38, wherein the system for 
recognizing traffic signs is provided with a data storage unit, 
in which at least one of the type, the position, the condition, 
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the visibility and the size of the recognized traffic sign is 
stored. 

43. A device according to Claim 38, wherein the display unit 
includes a signal emitter which emits an acoustic or optical 
signal when said vehicle travels along a previously traveled road 
segment and said system for recognizing traffic signs detects a 
change with respect to the position or the presence or absence of 
a traffic sign. 

44. A device according to Claim 38, wherein a program for 
utilization of a decision logic is provided in the system, which 
program displays the image of a traffic sign until a 
predetermined path length, as determined by at least one of (a) 
the map-based navigation system and (b) the traffic information 
system, has been traveled. 

45. A device as in Claim 44, wherein said predetermined path 
length is specific to the class of traffic sign. 

46. A device according to Claim 44, wherein a memory unit is 
provided in the system, in which the predetermined path lengths 
are stored and via which they are made available for processing. 

47. A device according to Claim 38, wherein a processing unit is 
provided in the system for recognition of traffic signs, which 
processing unit includes a program according to which it examines 
image data supplied by an image sensor for the presence of 
traffic signs, then extracts these data, separates these into 
upper and lower classes and in this context extracts class 
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specific characterizing data and separately supplies these to a 
classifier. 

48. A device according to Claim 47, wherein this classifier is 
constructed hierarchically, in multiple classification steps. 

49. A device according to Claim 48, wherein the classification 
steps are so constructed, that the classifier accomplishes the 
following functionalities: 

a) first, only the characterizing data associated 
with the upper class of the object is supplied to a 
classifier, 

aa) upon successful classification, wherein the class 
is recognized correctly with a high degree of 
confidence, the characterizing data are replaced by the 
appropriate symbolic representation of the upper class 
stored in the memory unit, 

ab) upon unsuccessful classification, wherein the 
class could not be correctly recognized with a high 
degree of confidence, the characterizing data for the 
upper class and the charactering data for the lower 
class are replaced by the corresponding original image 
data originating from the image sensor, 

b) if the classification of the upper class was 
successful, then subsequently the characterizing data 
associated with the lower class of the object are supplied 
to a classifier, 

ba) upon successful classification, wherein the class 
is recognized with a high degree of confidence, the 
characterizing data are replaced by the appropriate 
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symbolic representation of the lower class stored in 
the memory unit, 

bb) upon unsuccessful classification, wherein the 
class could not be correctly recognized with a high 
degree of confidence) the characterizing data for the 
lower class are substituted by the original image data 
from the image sensor. -- 



The specification and claims have been amended to conform 
the original translated specification and claims to U.S. 
requirements, i.e., appropriate section headers are added, 
reference in the specification to the claims have been amended in 
order to eliminate multiple dependent claims and claims 
improperly depending from multiple dependent claims, and to 
otherwise conform the claims to U.S. practice. Care has been 
taken to ensure that no new matter is added to the text. 

Entry and favorable consideration prior to consideration are 
respectfully requested. 



REMARKS 



PENDORF & CUTLIFF 
P.O. Box 20445 
Tampa, Florida 33622-0445 
(813) 886-6085 




Date: May 14, 2001 
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Honorable Commissioner of 
Patents and Trademarks 
Washington, D.C. 20231 

Sir: 

I, Stephan A, Pendorf , declare and state the following: 

I am a citizen of the United States residing at 3940 
Venetian Way, Tampa, Florida; 

I have lived in Germany for 14 years and am familiar with 
both the German and English languages and have experience as a 
technical translator; 

The attached English- language document is a full, true and 
faithful translation made by me of the text PCT Application No, : 
PCT/DE99/03627. 

The attached English- language document is a full, true and 
faithful translation made by me of the Substitute Pages for PCT 
Application No.: PCT/DE99/03627 . 

I hereby declare that all statements made herein of my own 
knowledge are true and that all statements made on information 
and belief are believed to be true; and further that these 
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METHOD FOR INCREASING THE PERFORMANCE OF A 
TRAFFIC SIGN RECOGNITION SYSTEM 



[0001] The invention concerns a process for improving the 
performance of a system for recognizing and/or indicating traffic 
signs according to the precharacterizing portion of Patent Claims 
1 and 10. 

[0002] Documents DE 3619824 C2 and DE 198 52 631 Al disclose 
processes and devices which recognize, classify and cause to be 
displayed traffic signs extracted from images of traffic scenes. 
Herein the processes analyze the image data provided by image 
sensors without any pre -recognition regarding the actual 
scenario. For this reason these known processes are time 
consuming and require a high computer capacity and performance in 
order to be able to process this data in real time. 

[0003] The task of the invention is thus to provide a process and a 
device, which enhances the performance of previously known 
systems for recognizing and displaying the contents of traffic 
signs. To accomplish this, on the one hand, the computer 
capacity and performance required for the real time operation 
must be reduced and, on the other hand, the display quality of 
the information contents of recognized traffic signs must be 
improved . 

[0004] The task is solved by a process and a device, in which a 
system for recognizing and/or indicating traffic signals, the 
system including a camera and a therewith associated evaluation 
unit for image recognition and/or for image display, utilizes a 
map-based navigation system and/or information originating from a 
traffic information system in the recognition and/or the display 
of the content of traffic signs. 
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[0005] In one conceivable embodiment of the invention the 
communication between the system for recognizing and/or display 
of traffic signs and the map-based navigation system or as the 
case may be the traffic information system occurs via a vehicle 
on-board data bus. 

[0006] Preferably, the system for detection of traffic signs can be 
switched to enhanced performance when, on the basis of 
information as could be supplied by the map-based navigation 
system and/or the traffic information system, it can be 
determined that the vehicle is passing through an area in which a 
reliable detection of traffic signs may be problematic. Examples 
of such areas in which caution is to be taken include transition 
areas for entry and departure of residential areas. There signs 
can be found indicating entry of a residential area and departure 
from a residential area, which imply speed limitations or, as the 
case may be, lifting of the speed limitations. These signs are 
however not round as conventional, but rather square. The signs 
are easily confused with commercial signs, particularly when 
traffic sign recognition is carried out using monochrome cameras, 
since here the relatively significant yellow coloration of the 
signs cannot be used as a recognition criteria. A further area 
in which particular care is to be taken is the environment of 
traffic inf luencing-f acilities and change or detour signs. Here 
one must presume special display and arrangement of the traffic 
signs, such as for example an intense representation of the black 
and white parts of the traffic sign as provided for improved 
night visibility. There is also an increased possibility of a 
suddenly occurring change or switching on of a traffic sign 
symbol, for example, sudden speed reduction to 120 km/h on 
highways. A particular type of traffic sign is a traffic light. 
In principle these can be classified with signs which change, 
since the condition of the display can change and therewith an 
increased observance is required. Using map-based information it 
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is however possible to operate the system with increased 
recognition performance for recognition of traffic signs in areas 
of traffic lights (intersections, train crossings) . 



[0007] While the above examples were concerned principally only 
with interaction with information from map-based navigation 
systems, examples of situations will be presented below in which 
increased recognition performance is required and which are 
advantageously associated with information originating from 
traffic information systems. This is essentially information 
advising that poor visibility due to fog or rain exists in 
certain regions traveled by the vehicle. In such a situation 
there is a presumption of increasingly poor contrast in the image 
data, so that the system for traffic sign recognition preferably 
must enhance the contrast improvement of the images. Preferably 
information regarding traffic jams is also taken into 
consideration, since here advanced notification can be given to 
the observer, in the form of the display of symbols of a traffic 
jam, a useful distance in advance of the traffic jam, without 
even having to pass such a traffic sign. 

[0008] By this inventive possibility of temporarily varying the 
performance of the traffic sign recognition system, this opens 
the possibility of economically realizing a system for 
recognition of traffic signs. It is also advantageous that it is 
not necessary to provide a system, which in normal operating 
conditions is inadequately tasked and only in critical situations 
is completely occupied. Rather, the possibility is presented of 
using modern multi-processor data processing units taking 
advantage of the internal vehicle data communications and the 
number of available data processors present in modern vehicles to 
utilize the temporarily free, unutilized computer capacity in 
productive economic employment . 
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[0009] A further advantageous design characteristic of the 
invention is comprised therein, that information regarding the 
type and/or the position and/or the condition and/or the 
visibility and/or the size of a recognized traffic sign can be 
stored in an data storage unit. This data storage unit can 
either be associated with a map-based navigation system or, as 
the case may be, a traffic information system, or alternatively 
may be a component of the system for traffic sign recognition 
and/or display. By the integration of this data storage unit the 
observer can be alerted, via an acoustic or optical signal, to 
the fact that along a previously traveled path a change in 
respect to the position or the presence or absence of a vehicle 
sign has occurred. Thereby it is possible to alert the observer 
to, for example, construction-site-specific traffic regulations 
or, in the case of changed right-of-way, regulations in a 
crossing situation. It is further of advantage when, in the same 
manner, a warning signal is provided to the observer when the 
system for traffic sign recognition does not recognize an actual 
traffic sign or cannot classify it adequately. 

[00010] The invention further provides benefits, when on the basis 
of the map-based position determination the possibility is 
presented to initiate or allow a program for use of a decision 
logic which limits the display a traffic sign for such a time, 
until a predetermined road distance has been traveled. This 
predetermined road distance is preferably classification specific 
and can be supplied using a storage unit or hard drive. For 
example in Germany the changing of streets is implicitly 
associated with the lifting of restrictions, such as speed limit 
restrictions. In addition, speed limits must be repeated at 
least every 3 km otherwise they are presumed to be rescinded. By 
this characteristic according to the invention it is achieved 
that a traffic sign can be displayed only as long as absolutely 
necessary even without explicit rescinding. 
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[00011] The invention has particularly advantageous benefits in 
combination with a system for recognition of traffic signs, in 
which the image data of the image sensor is analyzed and 
classified in an information-processing unit and, on the basis of 
the result, is stored in memory as image data representing class- 
specific image segments and/or symbolic figures, which are then 
displayed by means of the display unit. Therein, in the 
framework of the analysis by the information-processing unit, it 
is determined whether the image data of the image sensor contains 
one or more objects, which with sufficiently high probability are 
traffic signs. Consequently the so-detected objects are 
forwarded to a further processing and classification step. In a 
particularly advantageous manner, in the framework of the 
invention, during the further processing of the image data of the 
objects, a separation is undertaken into the traffic sign 
specific upper classes and lower classes. Thereby appropriate 
classification-specific characteristic data are extracted and 
relayed to a separate classification. Depending on whether the 
classification specific characteristic data are correctly 
recognized with high probability during the classification, they 
are replaced by the corresponding or appropriate symbolic image 
representing image data stored in the storage unit or by the 
appropriate original image data stemming from the image sensor. 
The image data resulting from the dissections are recombined into 
a synthetic image of a traffic sign and this image is then stored 
in the memory unit and brought to the display by means of the 
display unit. 

[00012] An important advantage of the invention is comprised 
therein, that it is no longer necessary to carry out the computer 
taxing and with high error quotient associated problem of 
classification of a relatively complex traffic sign which must be 
solved over its entirety, rather the problem, by separation of 
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the detected objects into traffic sign specific upper classes and 
lower classes in a preceding classification process, is dissected 
or redirected to clearer and therewith simpler classification 
problems. Therewith in like measure the demands on the necessary 
computing power and the image quality are reduced. 

[00013] A brief example will now be given using a speed limit sign 
for the separation of the traffic sign into traffic sign specific 
upper classes and lower classes. The traffic sign specific upper 
class of this traffic sign belongs to the class of regulating 
signs, which have a characteristic round shape with an internal 
symbol therein. In this special cases of a traffic speed 
limiting sign, the integrated symbol is a number which is 
circumscribed by a broad, red ring. In the case of this traffic 
speed limit indicating sign the internally lying symbol, that is 
the indication as to the speed to which a vehicle speed is to be 
limited, is an element of the traffic sign specific lower class. 

[00014] In the system for traffic sign recognition, with which 
advantageously the invention for improving the performance of a 
system for recognition and/or display of traffic signs can be 
combined, it should now be possible with comparatively low 
computer power and with low equipment expense (for example, a 
monochrome camera instead of a high resolution color camera) to 
check in real time whether the object of a particular shape 
within the detected or observed image is an actual traffic sign. 
If this is the case, then the classification specific 
characteristic data of the object are supplied to a separate 
classification for identification of the traffic sign. Generally 
a hierarchically organized classifier is employed for 
identification of the traffic sign. The classification occurs 
therein in multiple steps. For this, first the characteristic 
data or features associated with the upper class of the object 
are supplied to the classification. If a classification occurs 
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successfully (that is, the class could be correctly recognized 
with high confidence) , then the characteristic data are replaced 
by the appropriate symbolic representation of the object stored 
in the memory unit. If the classification is unsuccessful (that 
is, the class could not be recognized with high confidence), then 
the characteristic data of the upper class and the characterizing 
data of the lower class are replaced by the original image data 
originally stemming from the image sensor and the image 
classification process is ended. A further step within the 
classification procedure is initiated when the classification of 
the upper class was successful. In this case, the characterizing 
data belonging to the lower class of the object are subsequently 
supplied to a classifier. In the case of successful 

classification of this lower class (that is, the class could be 
recognized with high confidence) , then the characterizing data 
are replaced by the appropriate or corresponding symbolic 
representation of the lower class stored in the memory unit. On 
the other hand, in the case of unsuccessful classification (that 
is, the class could not be recognized with high confidence) the 
characterizing data of the lower class is replaced by the 
corresponding original image data stemming from the image sensor. 

[00015] With this hierarchical process a low error rate traffic 
sign recognition is provided. Therein both hierarchical steps 
for identification of the upper class or as the case may be the 
identification of the lower class may themselves again be 
hierarchically organized. With respect to the design of the 
classifiers employed within the hierarchical classification it is 
conceivable to so design these, that they make decisions on the 
basis of a comparison of the distance of the characterizing data 
supplied to them from the class typical characterizing data sets 
stored in the memory unit. Thereby the traffic sign identifying 
object is assigned to that respective upper class, to which the 
distance in the measurement space is the lowest. Thereby it is 
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possible to so define a tolerance threshold which is compared 
with a minimal distance value, wherein the distance value falling 
below the tolerance threshold serves as indicator to a high 
probability of correct occurrence of recognition. In another 
advantageous design it is also possible to so design the 
classifiers, that they can be trained with the aid of a traffic 
sign specific learning succession of passes. This process is 
particularly advantageous, since the quality of the image data 
delivered by the image sensor depends strongly upon the 
environmental factors such as weather and light relationships. 



- 8 - 



Patent Claims: 



1 . Process for improving performance of a system for 
recognizing and/or displaying traffic signs, which includes a 
camera and a therewith associated evaluation unit for image 
recognition and/or image display, thereby characterized, that the 
system utilizes information originating from at least one of a 
map-based navigation system and/or a traffic information system 
in the evaluation and/or display of contents of traffic signs, 

2. Process according to Claim 1, thereby characterized, that 
the system for recognition and/or display of traffic signs is in 
communication with the map-based navigation system and/or traffic 
information system via an onboard data bus. 

3. Process according to Claim 1 or 2, thereby characterized, 
that the system for traffic recognition begins to operate at an 
enhanced performance when it is determined on the basis of map- 
based information that the vehicle is passing through area in 
which recognition of traffic signs may be problematic. 

4. Process according to one of Claims 1 through 3, thereby 
characterized, that within problematic areas supplemental 
subdivided recognition processes are carried out, in such a 
manner that 

a) in the area of community entrances and community exits 
there is a specific search for signs indicating entry of a 
community and leaving of a community, 

b) in the area of traffic influencing facilities, changing 
signs or traffic lights, separately attention is paid to a 
change in the type and manner of the representation of the 
traffic sign, 

c) in an area, in which poor visibility due to fog or rain 
is present and a higher probability of contrast -poor images 
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must be calculated with, preferably an increase in the 
contrast enhancement of the image data is effected, 

5. Process according to one of the preceding claims, thereby 
characterized, that the type and/or the position and/or the 
condition and/or the visibility and/or the size of a recognized 
traffic sign is stored in a data storage unit associated with the 
map-based navigation system or, as the case may be, the traffic 
information system. 

6. Process according to one of the preceding claims, thereby 
characterized, that the type and/or the position and/or the 
condition and/or the visibility and/or the size of a recognized 
traffic sign is stored in a data storage unit associated with the 
system for recognition of traffic signs, 

7. Process according to one of Claims 5 or 6, thereby 
characterized, that by means of a signal producer contained in 
the display unit, an acoustic or optical signal is emitted when, 
along an already traveled road segment, a change with respect to 
the position or a change as to the presence or absence of a 
traffic sign occurs. 

8. Process according to one of the preceding claims, thereby 
characterized, that via a program stored in the storage unit a 
decision logic is controlled, via which the information 
processing unit controls the display unit in such a manner, that 
it displays the synthesized image of the traffic sign so long, 
until a predetermined traveled path stored in the storage unit is 
traveled, which is preferably class specific. 

9. Process according to Claim 8, thereby characterized, that 
the predetermined traveled paths are programmed into a data 
storage unit. 
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10. Process according to one of the preceding claims, thereby 
characterized, that the system for recognition of traffic signs, 
during the processing of image data provided by an image sensor, 
examines these for the presence of traffic signs, then extracts 
these, separates these into upper and lower classes and in 
association therewith extracts class-specific characterizing 
data, which are then supplied to a separate classification. 

11. Process according to Claim 10, thereby characterized, that 
the classification occurs hierarchically, in multiple 
classification steps. 

12. Process according to Claim 11, thereby characterized, that 
the classification steps are essentially set up follows: 

a) first, only the characterizing data associated 
with the upper class of the object is supplied to a 
classifier, 

aa) upon successful classification (i.e., the class 
could be recognized correctly with a high degree of 
confidence) the characterizing data are replaced by the 
appropriate symbolic representation of the upper class 
stored in the memory unit, 

ab) upon unsuccessful classification (i.e., the class 
could not be correctly recognized with a high degree of 
confidence) the characterizing data for the upper class 
and the charactering data for the lower class are 
replaced by the corresponding original image data 
originating from the image sensor, 

b) if the classification of the upper class was 
successful, then subsequently the characterizing data 
associated with the lower class of the object are supplied 
to a classifier, 
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ba) upon successful classification (i.e., the class 
could be recognized with a high degree of confidence) 
the characterizing data are replaced by the appropriate 
symbolic representation of the lower class stored in 
the memory unit, 

bb) upon unsuccessful classification (i.e., the class 
could not be correctly recognized with a high degree of 
confidence) the characterizing data for the lower class 
are substituted by the original image data from the 
image sensor, 

13 . Device for enhancing the performance of a system for 
recognizing and/or displaying traffic signs, which includes a 
camera and a therewith associated evaluation unit for image 
recognition and/or image display, thereby characterized, that the 
system is associated with at least one of a map-based navigation 
system and/or a traffic information system, in order to utilize 
information originating therefrom in the recognition and/or the 
display of contents of traffic signs. 

14. Device according to Claim 13, thereby characterized, that 
the connection between the system for recognizing and/or display 
of traffic signs and the map-based navigation system and/or the 
vehicle traffic information system is via an onboard data bus, 

15. Device according to Claim 13 or 14, thereby characterized, 
that the system for traffic sign recognition has the capacity to 
work with highened performance when, on the basis of map-based 
information, it is determined that the vehicle is passing through 
an area in which the recognition of traffic signs may be 
problematic . 

16. Device according to one of the preceding claims, thereby 
characterized, that the map -based navigation system or, as the 
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case may be, the traffic information system, is provided with a 
data storage unit, in which the type and/or the position and/or 
the condition and/or the visibility and/or the size of the 
recognized traffic sign is stored. 

17. Device according to one of the preceding claims, thereby 
characterized, that the system for recognizing of traffic signs 
is provided with a data storage unit, in which the type and/or 
the position and/or the condition and/or the visibility and/or 
the size of the recognized traffic sign is stored* 

18. Device according to one of Claims 16 or 17, thereby 
characterized, that the display unit includes a signal emitter 
which emits an acoustic or optical signal when along an already 
traveled road segment a change occurs with respect to the 
position or the presence or absence of a traffic sign. 

19. Device according to one of the preceding claims, thereby 
characterized, that a program for utilization of a decision logic 
is provided in the system for recognizing and/or displaying 
traffic signs, which displays the image of a traffic sign until a 
predetermined path length, based upon the map-based navigation 
system and/or the traffic information system, has been traveled, 
which is preferably class specific. 

20. Device according to Claim 16, thereby characterized, that a 
memory unit is provided in the system for recognition and/or 
providing traffic signs, in which the predetermined path lengths 
are stored and via which they are made available for processing. 

21. Device according to one of the preceding claims, thereby 
characterized, that a processing unit is provided in the system 
for recognition of traffic signs, which processing unit examines 
image data supplied by an image sensor for the presence of 
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traffic signs, then extracts these data, separates these into 
upper and lower classes and in this context extracts class 
specific characterizing data and separately supplies these to a 
classifier. 

22. Device according to Claim 21, thereby characterized, that 
this classifier is constructed hierarchically, in multiple 
classification steps. 

23. Device according to Claim 22, thereby characterized, that 
the classification steps are so constructed, that the classifier 
accomplishes the following functionalities: 

a) first, only the characterizing data associated with the 
upper class of the object is supplied to a classifier, 

aa) upon successful classification (i.e., the class 
could be recognized correctly with a high degree of 
confidence) the characterizing data are replaced by the 
appropriate symbolic representation of the upper class 
stored in the memory unit, 

ab) upon unsuccessful classification (i.e., the class 
could not be correctly recognized with a high degree of 
confidence) the characterizing data for the upper class 
and the charactering data for the lower class are 
replaced by the corresponding original image data 
originating from the image sensor, 

b) if the classification of the upper class was 
successful, then subsequently the characterizing data 
associated with the lower class of the object are supplied 
to a classifier, 

ba) upon successful classification (i.e., the class 
could be recognized with a high degree of confidence) 
the characterizing data are replaced by the appropriate 
symbolic representation of the lower class stored in 
the memory unit, 
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bb) upon unsuccessful classification (i.e., the class 
could not be correctly recognized with a high degree of 
confidence) the characterizing data for the lower class 
are substituted by the original image data from the 
image sensor. 
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ABSTRACT 



The invention relates to a method and device that increase the 
performance of prior art systems that recognize and display the 
contents of traffic signs. The invention reduces the amount of 
computer capacity required for real time operation and improves 
the display quality of the contents of known traffic signs. The 
system for recognizing and/or displaying traffic signs comprises 
a camera and an evaluation unit connected thereto for image 
recognition and/ or display. Said system uses information 
stemming from at least one map-based navigation system and/or one 
travel information system in order to recognize and/or display 
the contents of traffic signs. 
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PCT Substitute Page entered November 7, 2000 

METHOD FOR INCREASING THE PERFORMANCE OF A 
TRAFFIC SIGN RECOGNITION SYSTEM 

[0001] The invention concerns a process for improving the 
performance of a system for recognizing and/or indicating traffic 
signs according to the precharacterizing portion of Patent Claims 
1 and 10. 

[0002] Documents DE 3619824 C2 and DE 198 52 631 Al disclose 
processes and devices which recognize, classify and cause to be 
displayed traffic signs extracted from images of traffic scenes. 
Herein the processes analyze the image data provided by image 
sensors without any pre-recognition regarding the actual 
scenario. For this reason these known processes are time 
consuming and require a high computer capacity and performance in 
order to be able to process this data in real time. 

Japanese document JP 06348991 discloses a system and a process 
for recognizing traffic signs, wherein the system employs a color 
camera and a therewith associated evaluation unit for image 
recognition. The invention is concerned with the task of, by 
evaluating the color information from the image information, 
automatically recognizing lane boundaries and devices for 
regulating traffic. Herein however there is no supplementation 
of the image information from the color camera with information 
from a map-based navigation system or a traffic information 
system for enhancing the recognition performance of the system. 
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New Patent Claims (PCT Substitute Page entered November 7, 2000): 



1. Process for improving performance of a system for 
recognizing traffic signs, which includes a camera and a 
therewith associated evaluation unit for image recognition or for 
image display, thereby characterized, that the system utilizes 
information originating from at least one of a map-based 
navigation system or a traffic information system in the 
evaluation or display of contents of traffic signs. 

2. Process according to Claim 1, thereby characterized, that 
the system is in communication with the map-based navigation 
system or the traffic information system via an onboard data bus. 

3. Process according to Claim 1 or 2, thereby characterized, 
that the system for traffic sign recognition begins to operate at 
an enhanced performance when it is determined on the basis of 
map-based information that the vehicle is passing through area in 
which recognition of traffic signs may be problematic. 

4. Process according to one of Claims 1 through 3, thereby 
characterized, that within problematic areas supplemental 
subdivided recognition processes are carried out, in such a 
manner that 

a) in the area of community entrances and community exits 
there is a specific search for signs indicating entry of a 
community and leaving of a community, 

b) in the area of traffic influencing facilities, changing 
signs or traffic lights, separately attention is paid to a 
change in the type and manner of the representation of the 
traffic sign, 

c) in an area, in which poor visibility due to fog or rain 
is present and a higher probability of contrast -poor images 
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must be calculated with, preferably an increase in the 
contrast enhancement of the image data is effected* 

5. Process according to one of the preceding claims, thereby 
characterized, that the type or the position or the condition or 
the visibility or the size of a recognized traffic sign is stored 
in a data storage unit associated with the map-based navigation 
system or the traffic information system. 

6. Process according to one of the preceding claims, thereby 
characterized, that the type or the position or the condition or 
the visibility or the size of a recognized traffic sign is stored 
in a data storage unit associated with the system for recognition 
of traffic signs, 

7. Process according to one of Claims 5 or 6, thereby 
characterized, that by means of a signal producer contained in 
the display unit, an acoustic or optical signal is emitted when, 
along an already traveled road segment, a change with respect to 
the position or a change as to the presence or absence of a 
traffic sign occurs. 

8. Process according to one of the preceding claims, thereby 
characterized, that starting at the map-based navigation system 
or the traffic information system, a program for application of a 
decision logic is controlled in such a manner, that the image of 
the traffic sign is displayed for such a period, until a 
predetermined traveled path stored in the storage unit is 
traveled, which is preferably class specific. 

9. Process according to Claim 8, thereby characterized, that 
the predetermined traveled paths are programmed into a data 
storage unit. 
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10. Process according to one of the preceding claims, thereby 
characterized, that the system for recognition of traffic signs, 
during the processing of image data provided by an image sensor, 
examines these for the presence of traffic signs, then extracts 
these, separates these into upper and lower classes and in 
association therewith extracts class-specific characterizing 
data, which are then supplied to a separate classification. 

11. Process according to Claim 10, thereby characterized, that 
the classification occurs hierarchically, in multiple 
classification steps. 

12. Process according to Claim 11, thereby characterized, that 
the classification steps are essentially set up follows: 

a) first, only the characterizing data associated 
with the upper class of the object is supplied to a 
classifier, 

aa) upon successful classification, wherein the class 
could be recognized correctly with a high degree of 
confidence, the characterizing data are replaced by the 
appropriate symbolic representation of the upper class 
stored in the memory unit, 

ab) upon unsuccessful classification, wherein the 
class could not be correctly recognized with a high 
degree of confidence, the characterizing data for the 
upper class and the charactering data for the lower 
class are replaced by the corresponding original image 
data originating from the image sensor, 

b) if the classification of the upper class was 
successful, then subsequently the characterizing data 
associated with the lower class of the object are supplied 
to a classifier, 

ba) upon successful classification, wherein the class 
could be recognized with a high degree of confidence, 
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the characterizing data are replaced by the appropriate 
symbolic representation of the lower class stored in 
the memory unit, 

bb) upon unsuccessful classification, wherein the 
class could not be correctly recognized with a high 
degree of confidence, the characterizing data for the 
lower class are substituted by the original image data 
from the image sensor. 

13 . Device for enhancing the performance of a system for 
recognizing or displaying traffic signs, which includes a camera 
and a therewith associated evaluation unit for image recognition 
or for image display, thereby characterized, that the system is 
associated with at least one of a map-based navigation system or 
a traffic information system, in order to utilize information 
originating therefrom in the recognition and/or the display of 
contents of traffic signs. 

14. Device according to Claim 13, thereby characterized, that a 
connection is made via an onboard data bus between the system, 
the map-based navigation system and/or the vehicle traffic 
inf ormation system is . 

15. Device according to Claim 13 or 14, thereby characterized, 
that the system for traffic sign recognition has the capacity to 
work with highened performance when, on the basis of map -based 
information, it is determined that the vehicle is passing through 
an area in which the recognition of traffic signs may be 
problematic. 

16. Device according to one of the preceding claims, thereby 
characterized, that the map-based navigation system or, as the 
case may be, the traffic information system, is provided with a 
data storage unit, in which at least one of the type, the 
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position, the condition, the visibility and the size of the 
recognized traffic sign is stored. 



17. Device according to one of the preceding claims, thereby 
characterized, that the system for recognizing of traffic signs 
is provided with a data storage unit, in which at least one of 
the type, the position, the condition, the visibility and the 
size of the recognized traffic sign is stored. 

18. Device according to one of Claims 16 or 17, thereby 
characterized, that the display unit includes a signal emitter 
which emits an acoustic or optical signal when along an already 
traveled road segment a change occurs with respect to the 
position or the presence or absence of a traffic sign. 

19. Device according to one of the preceding claims, thereby 
characterized, that a program for utilization of a decision logic 
is provided in the system, which displays the image of a traffic 
sign until a predetermined path length, based upon the map-based 
navigation system or the traffic information system, has been 
traveled, which is preferably class specific. 

20. Device according to Claim 16, thereby characterized, that a 
memory unit is provided in the system, in which the predetermined 
path lengths are stored and via which they are made available for 
processing. 

21. Device according to one of the preceding claims, thereby 
characterized, that a processing unit is provided in the system 
for recognition of traffic signs, which processing unit examines 
image data supplied by an image sensor for the presence of 
traffic signs, then extracts these data, separates these into 
upper and lower classes and in this context extracts class 
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specific characterizing data and separately supplies these to a 
classifier. 

22. Device according to Claim 21, thereby characterized, that 
this classifier is constructed hierarchically, in multiple 
classification steps. 

23. Device according to Claim 22, thereby characterized, that 
the classification steps are so constructed, that the classifier 
accomplishes the following functionalities: 

a) first, only the characterizing data associated 
with the upper class of the object is supplied to a 
classifier, 

aa) upon successful classification, wherein the class 
could be recognized correctly with a high degree of 
confidence, the characterizing data are replaced by the 
appropriate symbolic representation of the upper class 
stored in the memory unit, 

ab) upon unsuccessful classification, wherein the 
class could not be correctly recognized with a high 
degree of confidence, the characterizing data for the 
upper class and the charactering data for the lower 
class are replaced by the corresponding original image 
data originating from the image sensor, 

b) if the classification of the upper class was 
successful, then subsequently the characterizing data 
associated with the lower class of the object are supplied 
to a classifier, 

ba) upon successful classification, wherein the class 
could be recognized with a high degree of confidence, 
the characterizing data are replaced by the appropriate 
symbolic representation of the lower class stored in 
the memory unit, 
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bb) upon unsuccessful classification, wherein the 
class could not be correctly recognized with a high 
degree of confidence, the characterizing data for the 
lower class are substituted by the original image data 
from the image sensor. 
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